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ABSTRACT 



A distributed data processing system includes a [durality of 
nodes interconnected by bidirectional communication links. 
Each^node includes a control message line for handling of 
control messages and a control memory for storing the 
control messages. Each node further includes data message 
line for handling of data messages and a data memory for 
storing the data messages. A processor in the node causes the 
data message line to queue and dispatch data messages from 
the data memoiy and the control message line to queue and 
dispatch control messages from the control memory. Each 
node includes N bidirectional communication links enabling 
the node to have at least twice as much input/ou^ut band- 
width as the control message line and data message line, 
combined. An input/output switch includes a routing pro- 
cessor and is coupled between the N bidirectional ccomnu- 
nication links, the data message line and control message 
line. The input/ou^ut switch dispatches either a control 
message or a dala^message over at least one of the bidirec- 
tional communicationrlisks in accordance with an output 
from the routing control processor, thereby enabling each 
communication link to carry either data or control messages. 
If a communication link is busy with either a control or a 
data message, the routing control processor increments to 
another conmmnication link to enable dispatch of a queued 
message. 

10 Claims, 5 Drawing Sheets 
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MULH-NODE NETWORK WITH wise stores the messages packets that cannot be routed 

INTERNODE SWITCHING PERFORMED because of circuit conflicts. The router is also able to meige 

WITHIN PROCESSOR NODES, EACH NODE message packets addressed to a same destination and to also 

SEPARATELY PROCESSING DATA AND broadcast message packets selectively to certain processor 

CONTROL MESSAGES 5 nodes in the nodal array, U.S. Pat No. 4,942^17 to Cok 

describes a tonis-like. distributed-memoiy, parallel com- 

This is a continuation of application Ser. No. 08/448,901 puter having rows of processors, with each fHOcessor having 

filed on May 24, 1995 now abandoned. an independent memory* A common input^ou^ut channel is 

ad^ted to be connected to a single row of processors by a 

FIELD OF THE INVENTION lo buffer associated with each processor in a row. 

This invention relates to a multi-node network that Notwithstanding the multiplicity of network interconnec- 

employs dis^ibuted switching of inter-node communication schemes in the prior art, none address in a satisfactory 

links and, more particularly, to a multi-node network manner the problem that occurs when long data messages 

wherein control and data messages are handled along sepa- transmitted and prevent transmission of control mes- 

rate paths within nodes. sages for the duration of the transmission. Further, within 

individual nodes of the network, it is vital that control 

BACKGROUND OF THE INVENTION messages be processed expeditiously without having to 

With the availability of low cost microprocessors, await the processing of data messages, 
distributed, multi-node data processors have come into Accordingly, it is an object of this invention to provide a 
vogue. Such systems include multiple nodes, each node ^ multi-nodal data processing system with an improved capa- 
induding a microprocessor. In certain systems, the nodes arc bility for handling data and control messages, 
interconnected by fixed circuits — and in others, the nodes Jt is another object of this invention to provide an 
are interconnected via centralized switches. Such systems improved multi-nodal data processing system which is con- 
are often applied to solution of large problems wherein each ^ figured to enable expeditious dispatch of control messages — 
node operates, in parallel, upon a different portion of the even in the presence of long duration data messages, 
problem, and the results arc combined in an ou^t node. it is yet another object of this invention to provide a 
Other multi-node systems are used to configure control multi-nodal data processing system wherein message rout- 
functions, such as controllers for disk drive memories. In ing circuitry is operable independently of a node's 
such systems, data transfers generally consist of long data ^ processor, thereby enabling intra-node processing to con- 
messages which include data from multiple tracks of a disk tinue even while handling message transfo^. 
drive. Control messages between nodes in such a system are 

relatively shcrt Control logic is provided in the respective SUMMARY OF THE INVENTION 

nodes to enable dispatch of control messages, notwithstand- A distributed data processing system includes a plurality 

ing the long data messages. However, within such nodes, of nodes interconnected by bidirectional communication 

both data and communication messages are handled through links. Each node includes control message line for handling 

common input/output (I/O) ports which often causes undue of control messages and a control memory for storing the 

delay in the processing of control messages. control messages. Each node further includes a data message 

The prior art describes a number of techniques for routing line f(x handling of data messages and a data menKMry for 
of data and control messages between nodes in a network. In ^ storing the data messages. A processor in the node causes the 
U.S. Pat No. 5,105,424 to Flaig et al., a distributed inter- data message line to queue and dispatch data messages from 
node message routing system is described wherein each the data memory and the control message line to queue and; 
node has separate routing control circuitry for each dimen- dispatch control messages from the control monory. Each 
sion of the network. In the Flaig et al. system, each node node includes N bidirectional communication links enabling 
adds routing directions to a message packet, dependent upon the node to have at least twice as much input/ou^ut band- 
the next destination of the packet Each packet is ttien width as tiie control message line and data message line, 
dispatched by the routing circuitry to a next adjacent node on combined. An input/ou^ut switch includes a routing pro- 
route to a destination node. At each node interface, a cesser and is coupled between the N bidirectional commu- 
mem(Hy map is stored of locations of other nodes in the nication links, the data message line and control message 
system and a corresponding route to each node. The routing ^ line. The input/output switch dispatdies either a control 
circuitry accesses the stored memory map to determine route message or a data message over at least one of the bidirec- 
information to be added to the message packet tional conununication links in accordance with an output 

U.S. Pat. No. 3,676,846 to Busch describes a centralized *c routing control processor, thereby enabling each 

message buffering system. Messages from low speed communication link to cany eithCT data or control messages, 

sources are buffered in a peripheral computer which tern- 55 ^ * communication link is busy with either a control or a 

porarily stores the incoming data and arranges it into strings message, the routing control processor increments to 

of data blodcs. The strings are then transmitted as messages another communication link to enable dispatch of a queued 

at a high data transfer rate to a central host computer: U.S. message. 

Pat No. 4380,046 to Ring describes a massively paraUel DESCRIPnON OF THE DRAWINGS 

processor system whertan paraUd data is processed and a eo ^x^v^i^xiv^i^ ixm ur^mni^yjo 

bidirectional data bus is employed to enable data transfers 1^0. 1 is a block diagram of a multi-node network that 

between individual processoxs. inc^lements the inventicxi hereof. 

U.S. Pat. No. 5,117,420 to HiUis et at describes a routing PIG. 2 is a block diagram of a representative node, 

technique for message packets wherein it is initially det^- FIG. 3 is a Hock diagram of an VO switch employed in 

mined if a message packet is addressed to a processing node 6S the node of FIG. 2. 

associated with the router. If not, the router directs the FIG. 4 is a block diagram of the conttol memory interface 

message packets to their destination, if possible, or other- block contained in the node of FIG. 2. 
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FIG. 5 is a block diagram of the data buffer intetf ace block The described node structure enables a distributed net- 

of the node of FIG. 2. work to be configured wherein all message switching fiinc- 

tions are distributed throughout the nodes and eliminates a 

DETAILED DESC3UFnON OF THE need for centralized switching control It is only within a 

INVENTION 5 node that control and data messages are segregated and 

The block diagram of HG. 1 illustrates a multi-node ^^^^ substantially independendy. Each I/O switch 42 

network 10 configured as a disk drive controller array. handles control and data messages in the same manner. 

Nodes A and D are data storage nodes that connect to ^^c^- multiple inter nodal links provide the system 

coupled disk drives 12. 14 and 16, 18, respectively. A pair of * ^^^^^ of robustness and redundancy in the event 

communication interface nodes B and C provide input/ ^ ^ 

ou^ut functions to coupled hostprocessors which make use Referring now to FIG. 3, a block diagram illustrates 

of the data storage facilities of the multi-node network, A further details of I/O switch 42. Each of communication 

cache node E jwovidcs teinporary storage facilities for both liiiks 44 is bi-directional and interfaces with a link adapter 

input and output of data transfer functions between network 50. Note that there are five link adapters shown, each of 

10 and one or more of the host processors. Multi-node which enables full duplex communication witii anotiier 

network 10 is expandable by addition of further nodes, all of node. Each link ad^tor 50 is preferably a 100 megabyte/ 

which are interconnected by an internal communication second, full duplex transceiver and includes small first-in/ 

network 20. first-out buffers to handle asynchronously clocked data firom 

Each of nodes A-E is configured from a common node ^ "thcrrouting switdi 52 or from another node connected via 

arrangement shown in FIG. 2. Eadi node includes a node ^ ^ communication hnk 44. 

processor 22 that controls the ovaall functions of tiie node. A routing processor 54, in the case of outgoing messages. 

Each node further includes a control message "line" 24 for accepts a destination address from either control message 

receiving, storing and dispatching control messages and a line interface 58 or data message line interface 60. In 

data message "line" 26 for receiving, storing and dispatch- response, routing processor 54 accesses route table 56, 

ing data messages. Control message line 24 comprises a examines the state of routing switch 52 and then selects an 

control memoy interface module 27 and a dynamic random optimal link adapter 50 for transmission of the message (if 

access memory (DRAM) 28. Each data message line 26 the message is destined to a different node). If the message 

includes a data buffer interface module 30 and a DRAM 3Z is for this node, routing processor 54 selects either the 

Data buffer interface module 30 connects to a plurality of control or buffer lines , as the case may be. If routing 

device interfaces 34, 36, etc. which, in turn, provide inter- ^ processor 54 finds that all links are busy, a busy indication 

faces to associated disk drives 38, 40, etc. Control messages is returned to node processor 22. 

originating from control memory interface 26, processor 22, I/O switch 42 further includes a node set up circuit tiiat is 

etc. enable control of various node actions. coupled to control message line interface 58 and is 

If the node of FIG. 2 is utilized as a cache node, random 35 employed to initialize I/O switch 42 and to read out status 

access memory 32 in data message line 26 is larger in size information therefrom The primary set up function is the 

than if the node is configured as a communication interface initialization of route table 56. Route table 56 has an 

node or a data storage node. If the node is configured as a individual entry for every possible node address. Each node 

communication interface node, communication adders are address entry further specifies a primary link ad^tcr as 

included (not shown) which enable connection to external ^ being the shortest path to the node address and three 

host processor. alternate link adapters in case the primary link adapter is not 

As will become apparent fi^m the description below, the available. When a message is routed through an intermediate 

architecture of multi-node network 10 is configured to primary communication path is not selected, 

cffidentiy handle both long data messages that characteris- routing processor 54 selects the next alternate link adapter, 

ticaUy occur fromAo disk drives and a multipUdty of small 45 a° available link adapter is founder none are found, in 

control messages which are required to enable operation of ^hidi case an interrupt is generated, in the node which 

the network and individual nodes. Within each node, the originated the message. 

respective control and data message lines enable segregation When either a contrcd message appears in control message 

oi control and data messages and allow their independent line interface 58 or a data ix^sage appears in data message 

processing and transfer to an ii^ut/output switch 42 which so interface 60, routing processor 54 examiikes the desti- 

is provided in each node. I/O switch 42 indudes apparatus nation address and routes the message through an available 

which enables it to independenUy switch messages that link adapter, in accordance with entries in route table 56 and 

arrive on one or more of input links 44 and are destined f(^ busy/available information from routing switch 52. Note, 

another node. Each node is preferably provided with at least that within I/O switch 42, data and control messages are 

twice as many communication links 44 as there are lines, so 55 handled identically as independent messages that are ready 

as to provide each node with at least twice the communi- for transfer. Because I/O switch 42 indudes at least twice as 

cation bandwidth that is present within the node. much bandwidth as is available on both the control and 

Each of comiminication links 44 is connected to another buffer message line, message transfers arc accomplished 

node, thereby enabling messages to either be directly routed without undue delay. Thus, control messages are dispatched 

to a connected node or to be routed through a connected 60 withouthavingto wait for the conopletion of a very long data 

node to another node, as tiie case may be. I/O switch 42 can message whidi may be in the process of transmission (or 

direct messages from any communication link 44 to any reception). 

other communication link 44. I/O switch 42 is further able Routing processor 54 preferably accepts messages from 

to select, dynamically, the best communication link 46 to either control message line interface 58 or data message line 

use, given a current state of the various links. Each of 65 interface 60 in dependence upon whichever interface first 

communication links 44 is independent and is usable for indicates a message available for transmission. If a '*tie^ 

either data or control messages. occurs between a control and data message, routing proces- 
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SOT 54 will preferably txansmit the control message first— it executed messages are forwarded to message execution unit 

being a very short message when compared to the data 83 where they are executed Successor executed messages 

message. are received into memory addresses within DRAM 28 in a 

Routing switch 52 is a conventional 7x7 non blocking logical i^ut data structure that corresponds to a function 
circuit switch which enable communication b^een any of 5 required by code in a received control message header. At 

link sLdaptas 50, control message Hnc interface 58 and data ^ executed, 

message line interface 60. Routing processor 54 is adapted ]<> HG. 5, details of data buffer mtcrfacc module 

to transmit a message via routing switch 52 to one or more ^ will be descnT>ed Structo^ it is similar to control 

link adapters 50, Routing processor 54, in response to a ^^odait 27 but does not mdude a tme- 

, / J. I V f . 1 L J i_i in or-day clock, a message execution umt or a processor bus 

value (or values) mesent in a message s header, enables lO , ^/ r\ 

^ ^ ^ T * *u interface. Data buffer interface module 30 is constructed 

assignment of either a smgle link adapter to the message or . u iaa w-u i^t i ^ 

iSi V 1, J 4^ * 1- u 4*. IZ. around a switch 100 which enables communication b^een 

multiple hnk ad^tcrs, so as to acconmhsh a "wavefronr ^. . \^ a ^ ^ i-*.- a t^t^a-lm ^ it 

^ — «^ , tjjg vanous connected functionahues, A DRAM controller 

type message routing. Note that for any message received w^^j* -^^ w i ^ ^ 

' 1- iT^i *!. ♦ J J- -* J * * *u J 1^ enables access to data memory 32. Local control bus 

via a link adapter 50 that is directed to vet another node, .^^^a^.^ ^ t 

Ji rJ ^ZT^^^A ' ui V ""'^^"^ J'^' oiivmiw uwc, interface 104 mtcrconnects to contrd memory interface 

routmg processor 54 is able to accomphsh a rerouting of that r 

-41, ^ * • * * * u module 27 to enable receipt and transfer of control mcs- 

message without requinng access to or an mtetrapt to be u • ^ ui • j c 

* J •*u- 1 ^ , . *^ sages. Data bus interface 106 enables access to and from 

generated withm node processor 22. In such manner, mcs- . . ^ , . , j ^ • , 

u- ^ *. I • u * II device mterface modules 34, 36 and their respectively 

sage roubng wilhu. the multo-node Mtwork is substonhaUy ^ ^ ' module IM and 

accoinpl^hed wiAout m^acbon of node processor ^ arVconsSucted in much the 

2^— thereby enabling each node's processor 22 to conccn- » v^u-l .JTw- . "* ""^ 

tratc its activities on internal node ftinctions. f^"" «'«P«'»P«t P«t l»Pf of 

„ r miauax nwic luucuous. control memoir interface 27, however, the data stnictuies 

Referring to HO. 4 fiirAer details arc shown of control processed therein are more complex. Order execution unit 

memory interface module 27. A switch 70 enaUes coimnu- no executes hardware executed messages related to data 

mcatioc between various of the functionalities within con- message line functions. 

trdmemonr interface module 27. -raeseare^^ As a resute of the above described control and daU 

mterface 72, an order exeoition unit 74, a DRAM co.to)!lor ^ ^^^^ ^^^^ ^ ^ ^ ^ 

76, a local control tasj^teface 78 ouJ^tportlo^cM and „^ substantially indcpendentiy. Hus enables long 

ii^p<rtlogic»2.DRAMcontroller76enabksattaduaent data messages to be qu^ed md transmitted while control 

a random access memory 2« via an cperuid memory bus ^ ^ ^^^^ inmsmission in a 

concurrent fashion. As a result, there is no need to preempt 
Processor bus Interface 72 is an interface to node proccs- the |rocessing of long data messages for control messages, 
sor 22 and provides a path that enables memory fetches and prcen^tion requires substantial state saving, loading 
stores. Order execution unit 74 interprets, fetches and stores and reloading and constitutes a significant processing over- 
data to certain memory-registers as orders to be executed. jjcad fw data messages. For exaii4)lc, in transfer of a data 
Some but not all orders are executed in this unit A dcscrip- message from a disk drive, the transfff of the data message 
tion of certain of the OTdeis will be found bdow. In essence, between nodes would be predicted at least five times by 
acdex execution unit 32 enables expedited handling of cer- associated control messages. This would substantially 
tain orders, without specific involvement of node jwocessor reduce the effective bandwidtfi of the message, 

^* 40 Further, the system enables use of indefinitely long mes- 

A time-of-day dock 86 includes botfi a clock incrcmenter sage lengths and thus departs from prior art packet-type 

and clock conq>aratOT (not shown). Each timc-of-day dock switching systems whidi segregate messages into specific 

86 in each node is periodicaUy rcsyndironizcd with time of message segment lengths (to enable insertion of control 

day clocks in otiier nodes. Order execution unit 74 provides message packets). The indefinitely long data message length 

the facilities to set, read out and otherwise control time-of- js accomplished by allowing multiple fidds of a message to 

day dock 86. be concatenated into a single message. Each fidd includes a 

Local control bus interface 78 enables access between delimiter specifying its length and a cyclic redundancy 

node processor 22 and data buffer interface 30 (FIG. 1) and check character fcUpwing it This allows data to be trans- 

the various device interfaces 34, 36 . . . that are connected mitted in one message without exceeding the ecror detection 
to data buffer interface 30. Node processor 22, via local 50 strength of the CRC character. 

control bus 78 is able to perfosm a number of fun<^ons,Le., Control messages are preferably sub-divided into two 

fetch and store to data memory 32; issuance of orders to one classes. A control noessage is decoded as to dass within 

or more device interfaces 34, 36; transfer of data from input port logic iXKxIule 82. One class is messages that roust 

control message line 24 to data message line 26 and vice be processed by the node's processor and the other class is 
versa, etc. ^5 those messages which may be processed within the node's 

Ou^ut port logic 80 indudes a plurality of queues 90 hardware without requiring intervention of the node's pro- 

wherein messages of various levels of priority are queued, cesser. The latter dass of messages comprise lock messages 

awaiting transmission. Output port logic 80 attempts to and state extraction control messages. The lock messages 

transmit eadi message in a highest priority queue before allow one node to acquire or rdease one or more locks in 
proceeding to a next lower priority queue. A message can be 60 another node without involving the processor in that node, 

sent from output port logic 80 to input port logic 82 via This reduces response time and overhead for acquiring a 

control message line interface 58 of the same node, but such lode by a factor of at least two and allows the requesting 

messages are normally sem to other nodes via VO switch 42. node to mflnipnijitf the lock synchronously, i.e., without 

Input port logic 82 recdves control messages from contrd . dispatching another task, and substantially improves the 
message line interface 58 and briefly buffers them. It 63 responsiveness of &e systenL 

decodes the control messages according to class (Le.. hard- State acquisition messages are messages which allow one 

ware executed or processor executed), and hardware node to extract state information from another node without 
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involving that node's processor. An example of such a 
micssage is a control message that causes read out of a 
portion of data stored in a node's memcRy and sends it back 
to the requesting node. Such a message is processed directly 
in cs-der execution unit 74 within memoiy interface 26. One s 
such command is a '*read destination address** which causes 
a fetching of data from a memory location in the destination 
node. The message body comprises a multi-byte memory 
address and when the message is received at the destination 
node, order execution unit 74 in the destination node writes lo 
the data from the control message's body into the memory 
location specified in the message body. Redispatch of the 
message conveys the accessed data to the requesting mode. 
Anotiier control message is a **write multiple*' message 
wherein any number of data words are stored in consecutive 15 
memory locations. The control message's body consists of a 
memory address and one or more data words to be written, 
starting at the memory address. 

It should be understood that the foregoing description is 
only illustrative of the invention. Various alternatives and 20 
modifications can be devised by those skilled in the ait 
without departing from the invention. For instance, while in 
FIG. 2, DRAM memories 28 and 32 are shown as separate 
modules, in a physical implementation tiiey would prefer- 
ably be configured as two potions of a unitary memory 25 
module. Further, the functions of control memory interface 
27 and data buffer interface 30 are preferably combined into 
a unitary interface module. Accordingly, the present inven- 
tion is intended to embrace all such alternatives, modifica- 
tions and variances which fall within the scc^ of the 3^ 
^rpended claims. 

What is claimed is: 

1. A distrilxjtcd data processing system configured as a 
disk drive controller array, conq>rising: 
a plurality of disk drive controller nodes, each disk drive 3^ 
controller node directly connected to each of N other 
nodes via a bi-directional concununication linV^ where 
N>a4, each said node thereby having at least twice as 
much input/output bandwidth as internal message han- 
dling bandwidth, each disk drive controller node fur- ^ 
ther con^nising: 

control message means for substantially independent han- 
dling of control messages; 

control memory means coupled to said control message 
means for storing control messages; 45 

disk drive means; 

data message means for substantially indq>endent han- 
dling of data messages; 

data memory means cou{ded to said data message means ^ 
for storing data messages to and from said disk drive 
means; 

processor means coupled to each of the aforesaid means, 
for causing said data message means to assemble, 
queue and dispatch data messages from said data S5 
memoiy means and said control message means to 
assemble, queue and dispatch control messages from 
said control memory means, said data messages gen- 
erally being substantially longer than said control mes- 
sages; and 60 

input/ou^ut (I/O) switdi means coupled between said N 
bi-directional communication links and said data mes- 
sage means and control message means and including 
routing control means, fc^* utilizing said N bidirectional 
conununication links to enable concurrent dispatch of 65 
control messages and data messages in accord with 
outputs from said routing control means, said I/O 
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switch means employing said N bidirectional commu- 
nication links to enable dispatch of said control 
messages, without requiring delay thereof pending 
completion of dispatch of said data messages. 

2. Tlie distributed data processing system as recited in 
claim 1, wherein said routing control means is responsive to 
a message received via a bidirectional communication link, 
which message is addressed to another node, to direct said 
message to said another node without involving an operation 
of said i^ocessor means. 

3. The distributed data processing system as recited in 
claim 1, wherein said I/O switch means dispatches data 
messages and control messages in an OTdcr received from 
said data message means and control message means, 
respectively, and in a case when a data message and control 
message are received substantially simultaneously, dis- 
patches said control message before said data message. 

4. The distributed data {mx^sing system as recited in 
claim 1, wherein said routing control means comprises: 

table means including an entry for each destination node, 
said entry specifying a primary bidirectional commu- 
nication link to be allocated for messages addressed 
said destination node and plural alternate bidirectional 
coimnunication Unks to be employed in case said 
primary bidirectional communication link is busy. 

5. The distributed data processing system as recited in 
daim 1, wherein each address appended to a message for 
said destination node includes a value employed by said 
routing control means to determine whether said message to 
said destination node is to be transmitted over just one or 
plural ones of said bi-directional communication Hnks. 

6. The distributed data processing system as recited in 
claim 4, wherein said routing control means includes control 
means responsive to received message address header infor- 
mation to determine if said information indicates another 
node, and if yes, to access an entry in said table means 
coiresponding to said information and to enable dispatch of 
said message, all without involvement of said processor 
means. 

7. The distributed data processing system as recited in 
claim 1, wherein said processor means controls said data 
message means and data memory means to assemble indefi- 
nitely long data messages with interspersed cyclic redun- 
dancy codes (CRC) for error checking, each said CRC 
positioned at a position in said indefinitely along data 
messages which is within an error detection capability of 
said Q(C, an indefinitely long data message comprising 
header information followed by data as read from a track or 
tracks of a disk drive. 

8. The distributed data processing system as recited in 
daim 1, fiirther oon^sing: 

means in said control message means for determining 
whether a received control message is a first type to be 
processed by said processor means or a second type to 
be iMX)cessed by an order execution unit; and 

an order execution unit for receiving second type control 
messages and for performing an operation required by 
said second type control message without intervention 
of said processor means. 

9. The distributed data processing system as redted in 
daim 8, wherein said second type control message requests 
a state report and said order execution unit retrieves a state 
value, inserts said state value into a response control mes- 
sage and dispatches said response control message. 

10. The distributed data processing system as redted in 
daim 1, wherein said control memory means and data 
memory means comprise two portions of a unitary random 
access memory. 

>^ * * * « 
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